Skip to content

fix: restart processLoop and leader pump on error with exponential backoff#66

Merged
jbiskur merged 1 commit intomainfrom
fix/restart-on-pump-error
Mar 31, 2026
Merged

fix: restart processLoop and leader pump on error with exponential backoff#66
jbiskur merged 1 commit intomainfrom
fix/restart-on-pump-error

Conversation

@jbiskur
Copy link
Copy Markdown
Contributor

@jbiskur jbiskur commented Mar 31, 2026

Summary

  • Individual flow type pumps die silently on error and are never restarted, causing permanent event processing stalls
  • Adds automatic restart with exponential backoff (1s → 2s → 4s → ... → 30s cap) to:
    • processLoop in data-pump.ts: restarts on unhandled rejection, resets backoff counter on successful acknowledge
    • Leader pump in data-pump-cluster.ts: restarts on start() failure, resets on successful event distribution
  • Guards prevent restart after stop() or leader step-down

Test plan

  • New tests verify backoff formula (1s, 2s, 4s, 8s, 16s, 30s cap)
  • New test verifies no restart after stop()
  • All existing tests pass
  • deno lint and deno fmt --check clean

Related: Usable fragment b6781bb9-c708-4788-9970-ef971904d44a (pump stall incident)

🤖 Generated with Claude Code

…ckoff

Individual flow type pumps that encounter errors die silently and are never
restarted, causing event processing to stall permanently. This adds automatic
restart with exponential backoff (1s-30s) to:

- processLoop: restarts on unhandled rejection, resets backoff on successful ack
- leader pump in cluster mode: restarts on start() failure, resets on successful distribution

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jbiskur jbiskur merged commit 980770f into main Mar 31, 2026
2 checks passed
@jbiskur jbiskur deleted the fix/restart-on-pump-error branch March 31, 2026 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant